perf: increase job concurrency, add channel routing and identity keys by kneckinator · Pull Request #151 · OpenSPP/OpenSPP2

kneckinator · 2026-04-03T05:16:20Z

Summary

Increase parallel-safe channel limits (cycle, eligibility_manager, program_manager) from 1 to 4, now that INSERT ON CONFLICT from phase 7 makes concurrent writes safe
Add serial entitlement_approval channel (limit=1) for fund balance tracking that must remain serial
Add serial statistics_refresh channel (limit=1) to prevent concurrent refresh storms from completion handlers
Add identity_key to all async job dispatchers to prevent duplicate job submission on double-click

Changes

queue_data.xml: Bump cycle/eligibility/program channels to limit=4; add entitlement_approval and statistics_refresh channels at limit=1
cycle_manager_base.py: Add identity_key to check_eligibility, prepare_entitlements, add_beneficiaries dispatchers; route mark_*_as_done to statistics_refresh
program_manager.py: Add identity_key to enroll_eligible dispatcher; route mark_enroll_eligible_as_done to statistics_refresh
eligibility_manager.py: Add identity_key to import_registrants dispatcher; route mark_import_as_done to statistics_refresh
entitlement_manager_base.py: Route set_pending_validation, validate, cancel entitlements to entitlement_approval channel
entitlement_manager_cash.py: Route validate_entitlements to entitlement_approval channel
entitlement_manager_inkind.py: Route set_pending_validation and validate_entitlements to entitlement_approval channel

Context

Phase 9 of 9 in the spp_programs performance optimization effort. Rebased on current 19.0. Version bumped to 19.0.2.0.10.

Test plan

./scripts/test_single_module.sh spp_programs — 615 tests, 0 failures
pre-commit run --files <changed_files> — all checks pass
For async operations: test via UI with ODOO_INIT_MODULES=spp_programs docker compose --profile ui up -d

gemini-code-assist

Code Review

This pull request implements bulk membership creation for programs and cycles using raw SQL to improve performance and handle duplicates. It introduces context flags to skip expensive statistics recomputations during bulk operations, adds explicit refresh methods, refactors asynchronous job handling with specific channels and identity keys, and updates queue configurations. Feedback focuses on ensuring UTC timestamps in raw SQL queries and correcting a missing return value to align with method documentation.

gemini-code-assist · 2026-04-03T05:19:45Z

+            values = []
+            params = []
+            for v in batch:
+                values.append("(%s, %s, %s, %s, %s, %s, now(), now())")


When using raw SQL in Odoo, it is recommended to explicitly use (now() at time zone 'utc') for create_date and write_date. This ensures that the timestamps are stored in UTC regardless of the database session's timezone configuration, adhering to Odoo's standard data storage practices.

Suggested change

values.append("(%s, %s, %s, %s, %s, %s, now(), now())")

values.append("(%s, %s, %s, %s, %s, %s, now() at time zone 'utc', now() at time zone 'utc')")

gemini-code-assist · 2026-04-03T05:19:45Z

+            for v in batch:
+                state = v.get("state", "draft")
+                enrollment_date = now if state == "enrolled" else None
+                values.append("(%s, %s, %s, %s, %s, %s, now(), now())")


When using raw SQL in Odoo, it is recommended to explicitly use (now() at time zone 'utc') for create_date and write_date. This ensures that the timestamps are stored in UTC regardless of the database session's timezone configuration, adhering to Odoo's standard data storage practices.

Suggested change

values.append("(%s, %s, %s, %s, %s, %s, now(), now())")

values.append("(%s, %s, %s, %s, %s, %s, now() at time zone 'utc', now() at time zone 'utc')")

gemini-code-assist · 2026-04-03T05:19:45Z

+            }
+            for partner_id in beneficiaries
+        ]
+        self.env["spp.cycle.membership"].bulk_create_memberships(vals_list, skip_duplicates=True)


The method _add_beneficiaries is missing a return statement. According to its docstring, it should return an integer representing the count of inserted members. Returning the result of bulk_create_memberships will satisfy this requirement.

Suggested change

self.env["spp.cycle.membership"].bulk_create_memberships(vals_list, skip_duplicates=True)

return self.env["spp.cycle.membership"].bulk_create_memberships(vals_list, skip_duplicates=True)

codecov · 2026-04-03T05:20:20Z

Codecov Report

❌ Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.45%. Comparing base (1f4203c) to head (832c497).
⚠️ Report is 4 commits behind head on 19.0.

Files with missing lines	Patch %	Lines
...ograms/models/managers/entitlement_manager_base.py	50.00%	1 Missing ⚠️
...rams/models/managers/entitlement_manager_inkind.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             19.0     #151      +/-   ##
==========================================
+ Coverage   71.36%   71.45%   +0.09%     
==========================================
  Files         932      932              
  Lines       54792    54794       +2     
==========================================
+ Hits        39101    39154      +53     
+ Misses      15691    15640      -51

Flag	Coverage Δ
spp_api_v2	`80.10% <ø> (ø)`
spp_api_v2_change_request	`66.85% <ø> (ø)`
spp_api_v2_cycles	`71.12% <ø> (ø)`
spp_api_v2_data	`64.41% <ø> (ø)`
spp_api_v2_entitlements	`70.19% <ø> (ø)`
spp_api_v2_gis	`71.52% <ø> (ø)`
spp_api_v2_products	`66.27% <ø> (ø)`
spp_api_v2_service_points	`70.94% <ø> (ø)`
spp_api_v2_simulation	`71.12% <ø> (ø)`
spp_api_v2_vocabulary	`57.26% <ø> (ø)`
spp_audit	`72.60% <ø> (+0.06%)`	⬆️
spp_base_common	`90.26% <ø> (ø)`
spp_case_entitlements	`97.61% <ø> (ø)`
spp_case_programs	`97.14% <ø> (ø)`
spp_cel_event	`85.11% <ø> (ø)`
spp_claim_169	`58.11% <ø> (ø)`
spp_dci_client_dr	`55.87% <ø> (ø)`
spp_dci_client_ibr	`60.17% <ø> (ø)`
spp_programs	`64.33% <80.00%> (+0.73%)`	⬆️
spp_security	`66.66% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
spp_programs/__manifest__.py	`0.00% <ø> (ø)`
spp_programs/models/managers/cycle_manager_base.py	`70.23% <100.00%> (+3.86%)`	⬆️
...pp_programs/models/managers/eligibility_manager.py	`83.73% <100.00%> (+8.13%)`	⬆️
...ograms/models/managers/entitlement_manager_cash.py	`63.31% <100.00%> (+4.52%)`	⬆️
spp_programs/models/managers/program_manager.py	`92.74% <100.00%> (ø)`
...ograms/models/managers/entitlement_manager_base.py	`72.34% <50.00%> (+8.08%)`	⬆️
...rams/models/managers/entitlement_manager_inkind.py	`55.42% <0.00%> (ø)`

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Increase parallel-safe channel limits from 1 to 4 (cycle, eligibility_manager, program_manager) now that INSERT ON CONFLICT makes these operations safe for concurrent execution. Add two serial channels: - entitlement_approval (limit=1): fund balance tracking must be serial - statistics_refresh (limit=1): avoid concurrent refresh storms Route entitlement approval/validation jobs to entitlement_approval channel. Route all completion handlers (mark_*_as_done) to statistics_refresh channel. Add identity_key to all async dispatch methods to prevent duplicate job submission when users double-click action buttons.

…patchers

gemini-code-assist Bot reviewed Apr 3, 2026

View reviewed changes

kneckinator assigned gonzalesedwin1123 Apr 10, 2026

kneckinator requested a review from gonzalesedwin1123 April 10, 2026 03:51

gonzalesedwin1123 approved these changes Apr 15, 2026

View reviewed changes

gonzalesedwin1123 force-pushed the worktree-perf+phase9-concurrency branch from 38bc517 to 0cacf62 Compare April 17, 2026 03:21

gonzalesedwin1123 added 2 commits April 17, 2026 11:23

docs: bump version to 19.0.2.0.10, add changelog for concurrency tuning

c0af41c

test: add coverage for channel routing and identity keys in async dis…

832c497

…patchers

gonzalesedwin1123 merged commit b061135 into 19.0 Apr 17, 2026
35 checks passed

gonzalesedwin1123 deleted the worktree-perf+phase9-concurrency branch April 17, 2026 04:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: increase job concurrency, add channel routing and identity keys#151

perf: increase job concurrency, add channel routing and identity keys#151
gonzalesedwin1123 merged 3 commits into19.0from
worktree-perf+phase9-concurrency

kneckinator commented Apr 3, 2026 •

edited by gonzalesedwin1123

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Apr 3, 2026

Uh oh!

gemini-code-assist Bot Apr 3, 2026

Uh oh!

gemini-code-assist Bot Apr 3, 2026

Uh oh!

codecov Bot commented Apr 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	values.append("(%s, %s, %s, %s, %s, %s, now(), now())")
	values.append("(%s, %s, %s, %s, %s, %s, now() at time zone 'utc', now() at time zone 'utc')")

	self.env["spp.cycle.membership"].bulk_create_memberships(vals_list, skip_duplicates=True)
	return self.env["spp.cycle.membership"].bulk_create_memberships(vals_list, skip_duplicates=True)

Conversation

kneckinator commented Apr 3, 2026 • edited by gonzalesedwin1123 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Context

Test plan

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kneckinator commented Apr 3, 2026 •

edited by gonzalesedwin1123

Loading

codecov Bot commented Apr 3, 2026 •

edited

Loading